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Abstract 

In this paper we show that during the retrieval process in a bi- 
nary symmetric Hebb neural network, spatial localized states can be 
observed when the connectivity of the network is distance-dependent 
and constraint on the activity of the network is imposed, which forces 
different level of activity in the retrieval and learning states. This 
asymmetry in the activity during the retrieval and learning is found 
to be sufficient condition in order to observe spatial localized retrieval 
states. The result is confirmed analytically and by simulation. Key- 
words: neural networks, spatial localized states, replica formalism 
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1 Introduction 



In a recent publication [Q it was shown that using linear-threshold 
model neurons, Hebb learning rule, sparse coding and distance-dependent 
asymmetric connectivity, spatial asymmetric retrieval states (SAS) 
can be observed. This asymmetric states are characterized by a spatial 
localization of the activity of the neurons, described by the formation 
of local bumps. 

Similar results have been reported in the case of Hebb binary model 
for associative neural network The observation is intriguing, be- 
cause all components of the network are intrinsically symmetric with 
respect to the positions of the neurons and the retrieved state is clearly 
asymmetric. An illustration of SAS in binary network is presented in 
FigEl 

In parallel to the present investigation, extensive computer simu- 
lations have been performed in the case of integrate and fire neurons 
[3j, where bump formations were also reported. These results are 
in agreement with our previous results .2, that networks of binary 
neurons do not show bumpy retrieval solutions when the stored and 
retrieved patterns have the same mean activity. 

The biological reason for this phenomenon is based on the transient 
synchrony that leads to recruitment of cells into local bumps, followed 
by desynchronized activity within the group 

When the network is sufficiently diluted, say less then 5%, then 
the differences between asymmetric and symmetric connectivity are 
minimal 6 . For this reason we expect that the impact of the asym- 
metrical connectivity will be minimal and the asymmetry could not 
be considered as necessary condition for the existence of SAS, as it is 
confirmed in the present work. 

There are several factors that possibly contribute to the SAS in 
model network. 

In order to introduce the spatial effects in neural networks(NN), 
one essentially needs distance measures and topology between the 
neurons, imposing some distribution on the connections, dependent 
on that topology and distances. The major factor to observe spatial 
asymmetric activity is of course the spatially dependent connectiv- 
ity of the network. Actually this is an essential condition, because 
given a network with SAS, by applying random permutation to the 
enumeration of the neurons, one will obviously achieve states without 
SAS. Therefore, the topology of the connections must depend on the 
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distance between the neurons. 

Due to these arguments, a symmetric and distance-dependent con- 
nectivity for all neurons is chosen in this study. 

We consider an attractor NN model of Hebbian type formed by 
N binary neurons {Si}, Si £ {— 1,1}, i = 1,...,N, storing p binary 
patterns £ {l...p}, and we assume a symmetric connectivity 

between the neurons Cy = Cji £ {0,1}, ca = 0. Cjj = 1 means that 
neurons i and j are connected. We regard only connectivities in which 
the fluctuations between the individual connectivity are small, e.g. 
Vj Ylj °ij ~ where c is the mean connectivity. 

The learned patterns are drawn from the following distribution: 

P(0 = ^ W - 1 + a) + ^ W + 1 + «), 

where the parameter a is the sparsity of the code. Note that the 
notation is a little bit different from the usual one. By changing the 
variables rj — ► £+a and substituting in the above equation, one obtains 
the usual form for the pattern distribution in the case of sparse code 

0- 

Further in this article we show that imposing symmetry between 
the retrieval and the learning states, i.e. equal probability distribu- 
tions of the patterns and the network activities, no SAS exists. Spatial 
asymmetry can be observed only when asymmetry between the learn- 
ing and the retrieval states is imposed. 

Actually, by using binary network and symmetrically distributed 
patterns, the only asymmetry between the retrieval and the learning 
states that can be imposed, independent on the position of the neu- 
rons, is the total number of the neurons in a given state. Having in 
mind that there are only two possible states, this condition leads to a 
condition on the mean activity of the network. 

To impose a condition on the mean activity, we add an extra term 
H a to the Hamiltonian 

H a = NRj2Si/N. 

i 

This term actually favors states with lower total activity Yl% Si that is 
equivalent to decrease the number of active neurons, creating asym- 
metry between the learning and the retrieval states. If the value of 
the parameter R = 0, the corresponding model has been intensively 
studied since the classical results of Amit et al. |B] for symmetrical 
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code and Tsodyks, Feigel'man [7] for sparse code. These results show 
that the sparsities of the learned patterns and the retrieval states are 
the same and equal to aN. In the case R ^ 0, the energy of the system 
increases with the number of active neurons (Si = 1) and the term 
H a tends to limit the number of active neurons below aN. Roughly 
speaking, the parameter R can be interpreted as some kind of chemi- 
cal potential for the system to force a neuron in a state 5 = 1. Here 
we would like to mention that Roudi and Treves pQ have also stated 
that the activity of the linear-threshold network, they study, has to 
be constrained in order to have spatially asymmetric states. However, 
no explicit analysis was presented in order to explain the phenomenon 
and the condition is not shown to be sufficient one, probably because 
the use of more complicated linear threshold neurons diminishes the 
importance of this condition. Here we point out the importance of the 
constraint on the network level of activity and show that it is a suffi- 
cient condition for observation of spatial asymmetric states in binary 
Hebb neural network. 

The goal of this article is to find minimal necessary conditions 
where SAS can be observed. We start with general sparse code and 
sparse distance-dependent connectivity and using replica symmetry 
paradigm we find the equations for the order parameters. 

Then we study the solutions of these equations using some approx- 
imations and compare the analytical results with simulation. 

The conclusion, drawn in the last part, shows that only the asym- 
metry between the learning and the retrieval states is sufficient to 
observe SAS. 



2 Analytical analysis 
2.1 Connectivity matrix 

For the analytical analysis of the SAS states, we consider the decom- 

(k) 

position of the connectivity matrix Cj,- by its eigenvectors a] : 

c ij = ^X k a^af\ £af\?>=fe, (1) 

k i 

where A/% are the corresponding (positive) eigenvalues. The eigenvec- 
tors are ordered by its corresponding eigenvalues in decreasing 
order, e.g. 

Vaf\af ,k>l^X k < A,. (2) 
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Figure 1: The components of the first eigenvectors of the connectivity matrix, nor- 
malized by a square root of their corresponding eigenvalue, in order to eliminate the 
effect of the size of the network. The first eigenvector has a constant component, 
the second one-sine-like component. N = 6400, c = 320/N, A = 319.8, Ai = 285.4. 



For convenience we introduce also the parameters b^: 

b^aP^fc. (3) 

(k) i— i 

To get some intuition of what a!- look like, we plot in Fig. [I] the first 
two eigenvectors a and a 1 . 

For a wide variety of connectivities, the first three eigenvectors are 
approximately the following: 

4 0) = y/l/N, 4 1} = yfi/N cos(2tt k/N), af ] = fi/N 'sin(2vrA: / 'N) . 

(4) 

Further, the eigenvalue Ao is approximately the mean connectivity 
of the network per node, that is the mean number of connections per 
neuron Ao = cN. 

2.2 Hamiltonian 

Following the classical analysis of Amit et al. [H], we study binary 
Hopfield model |Hj 

Hint = — —jy ^2 Si£,i c ij£,jSj, (5) 
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where the matrix cy denotes the matrix of connectivity and we have 
assumed Hebb's rule of learning jlUj . 

In order to take into account the finite numbers of overlaps that 
condense macroscopically, we use Bogolyubov's method of quasi aver- 
ages 11 j. To this aim we introduce an external field, conjugate to a 
final number of patterns {£f}, u = 1,2, s by adding a term 

fffc = -X>"Effs (6) 

v=l i 

to the Hamiltonian. 

Finally, as we already mentioned in the Introduction, in order to 
impose some asymmetry in the neural network's states, we also add 
the term 



(7) 



where we used « 1. The equality is exact for equal degree network. 
The over line in eq.® means spatial averaging: (.j) = 

The whole Hamiltonian we are studying now is H = Hi n t+Hh+H a : 

H = -^H s ^''v^ s j - E h " E + NR $$- ( 8 ) 

ijfi v=\ % 

2.3 RS solution 

By using the "replica formalism" ^2], for the averaged free energy per 
neuron we get: 

/=Hm Jim -_L(((Z»»-1), (9) 

where ((...)) stands for the average over the pattern distribution ), 
n is the number of the replicas, which are later taken to zero and (3 is 
the inverse temperature. 

Following jSj, ^3], the replicated partition function is represented 
by decoupling the sites using an expansion of the connectivity matrix 
Cjj over its eigenvalues Xi,l = 1,...,M and eigenvectors a\ (eqQ): 

^EE(^)(^-) + 

ppl ij 



((Z n )) = e -P aNn / 2c { (Tr SP exp 

P E hU E ff-S? - 0RN E b1S?/N 



ip ip 



), (10) 
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with a = p/N being the storage capacity. 

Following the classical approach of Amit et al.jH] we introduce 
variables m^ k for each replica p and each eigenvalue and split the 
sums over the first s "condensed" patterns, labeled by the letter v 
and the remaining (infinite) p — s, over which an average and a later 
expansion over their corresponding parameters was done. 1 We have 
supposed that only a finite number of m v k are of order one and those 
are related to the largest eigenvalues 

As a next step, we introduce the order parameters (OP) 

q r = wwsf, 

where p, a = 1, n label the replica indexes. Note the role of the con- 
nectivity matrix on the OP q k ' a by the introduction of the parameters 

The introduction of the OP r£' CT , conjugate to q k ' a , and the use 
of the replica symmetry ansatz ^2] fn v p k = rn k^1k' 7 = Ik for p ^ a 
and r k ' a = r& for p ^ a, followed by a suitable linearization of the 
quadratic in 5-terms and the application of the saddle-point method 
[Hj, give the following final form for the free energy per neuron: 

1 .„ 9 . lv^ ; . 9 a 0(1 — a 2 ) v-^ a0(l — a 2 ) v-^ 
— a(l - a 2 ) + - Y,( m k) ~ E r ^k + E ^r k + 

^ E[ln(l - 0(1 ~ a V + 0(1 - a 2 )q k ) - (11) 
0(1 - a 2 )g fc (l - 0(1 - a 2 K + 0(1 - a 2 )^)- 1 ] - 





a 



In the last expression we have introduced the variables p, k = Xk/cN 
and we have used the fact that the average over a finite number of pat- 
ters £ u can be self-averaged (SJ. In our case however the self-averaging 
is more complicated in order to preserve the spatial dependence of the 
retrieved pattern. The detailed analysis will be given in a forthcoming 
publication [Tlj . 



1 Without loss of generality we can limit ourself to the case of only one pattern with 
macroscopic overlap (v — 1). The generalization to v > 1 is straightforward. 
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The equations for the OP r/%, m& and are respectively: 

<?fc(l - a 2 ) 



(l-/3(l-o 2 )(A*fc-g fc )) 



2 ' 



(12) 



and 



dze 2 



tanh/? z U - a 2 ) 5>,6$ + £ + Rb° (13) 



g fc = / ^^-(bf) 2 tanh 2 /3 ^a(l - a 2 ) £ Jtfj&j + £ m^ftj + | . (14) 

At T = 0, keeping Cfc = P(fJ,k — Qk) finite and limiting the above 
system only to the first two coefficients, the above equations read: 



m = 


1-a 2 


47T 


mi = 


, 1 


Co = 


1 


Ci = 





g((f>)d(t> 



I — a 
Air 



g(cp) smt 



''A- 



/>) sin 2 <; 
/U fc (l - a 2 ) 

[l_(l_ a 2 )Cfc] 2' 



(15) 

(16) 
(17) 
(18) 
(19) 



where 



5(0) = 


erf (x±) + erf (^2) 




(20) 


7c(0) = 


[(l + a)e- (:El)2 + (l-a)e- (:r: 


!)2 ]/[v^y] 


(21) 


X\ = 


[(1 — a)(mo + mi^/2/xi sin 0) 


+ 


(22) 


X2 = 


[(1 + a) (mo + m\\/2[i\ sin </;) 




(23) 


y = 


\/2a(l - a 2 )(r + 2/iiri sin 2 




(24) 



When we assume m\ = mi = ... = 0, R = 0, we obtain the result 
of Gardner ^3]. When additionally /ii = we obtain the result of 
Amit et al. |Sj. Of course, in this approximation no conclusion about 
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Figure 2: Left - Computation of mo, mi according to the eqs. (|15H9|) . The sparsity 
of the code is a = 0.2 and R = 0.57. The sharp bound is due to the limitation of 
the equations only to the first two terms of the OP m, r, C, as well as to the finite 
scale effects. The result of the simulations is represented on the right side. 



the size of the bump can be drawn, because the bump characteristic 
size is fixed to the characteristic size of a sine wave, that is always one 
half of the size of the network. 

The result of the numerical solution of the eas. () 1511111)) is show in 
Fig- El (left). The sharp bound of the phase transition is a result of 
taking into account just two terms k = 0, 1 and the lack of finite-size 
effects in the thermodynamic limit. 



3 Simulations 

In order to compare the results, we also performed computer simula- 
tions. To this aim we chose the network's topology to be a circular 
ring, with a distance measure 

\i — j\ = min(i — j + N mod N, j — i + N mod N) 

and used the same connectivity as in Ref . [l a with typical connectivity 
distance a T N: 



P{dj = 1) 



1 -{\i-j\/Nf/2„l 



2ira T N 



Here the parameter po is chosen to normalize the expression in the 
brackets. When a x is small enough, then spatial asymmetry is ex- 
pected. 
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Figure 3: Typical bump form. The figure shows the running average of the local 
overlap mu\ = Si£i vs the position of the neuron (run length 100 and 300). The 
sparsity of the code is a = 0.2, R is chosen to supply 0.5 of the unconstrained 
activity. N = 6400, c = 0.05. 



In order to measure the overlap between the states of the patterns 
and the neurons and the effect of the single-bump spatial activity 
during the simulation, we used the ratio mi/mo, where 

k 

and 

k 

Because the sine waves appear first, mi/mo results also to be a sensi- 
tive asymmetric measure, at least compared to the visual inspection 
of the running averages of the local overlap = An example 
of the local overlap form is given in Fig|21 

Let us note that m\ can be regarded as the power of the first 
Fourier component and mo can be regarded as a zeroth Fourier com- 
ponent, that is the power of the direct-current component. 

The corresponding behavior of the order parameters mo, m\ in the 
zero-temperature case, obtained by the simulations, is represented in 
Fig- El (right). Note the good correspondence between the numerical 
solution of the analytical results and the results obtained by simula- 
tion. 
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Figure 4: Left: Phase diagram R versus a for a = 0.001 (a/c = 0.02). The 
SAS region, where local bumps are observed is relatively large. The Z region 
corresponds to trivial solutions for the overlap. Right: SAS region for different 
values of a. High values of a limit the SAS region. Note that not all of the values 
of a are feasible with arbitrary sparsity of the network c. 

4 Discussion 

We have investigated the phase diagram when fixing the capacity 
taking into account the connectivity of the network. We have ob- 
served a stable area of solutions corresponding to the formation of 
local bumps, i.e. solutions with OP mi ^ for wide range of the 
parameters load a, sparsity a and the retrieval asymmetry parameter 
R (Fig^J). Note that a = p/N that is used in this paper, although 
natural when working with eigenvectors, is slightly different from the 
usual one a' = P/{cN) = a/c. The range of values of a/c corresponds 
to biologically plausible values. 

According to this phase diagram, there exist states with a = 
and spatial asymmetric retrieval states. Therefore, the sparsity of the 
code is not a necessary condition for SAS. Also it is clear that there 
is no necessity to use more complex model neuron than a binary one 
in order to observe SAS. It is worth mention that the sparsity of the 
code makes SAS better pronounced. 

On the other hand, there is no state with SAS and H a = 0, and 
therefore the asymmetry between the retrieval activity and the learned 
patterns is essential for the observation of the phenomenon. 

The diagram in Fig@] shows a clear phase transition with R. For 
small values of R, the effect (SAS) is not present. If R > 1, then the 
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Figure 5: The critical storage capacity a c as a function of R. a = 0.4 The drop 
of a c is clearly seen on the transition to SAS state. 



only stable state is the trivial Z state, as all the nonzero solutions are 
suppressed. Only for intermediate values of R, the bumpiness occurs. 
The load a of the network shrinks the area of parameters where bumps 
are observed. However, the area remains large enough for all values 
of a. 

Our observations show that the parameter R is a very rigid one in 
the sense that the behavior of the system changes drastically for small 
changes of R. Richer behavior can be obtained when the term H a is 
replaced by a quadratic term of the total activity. Detailed analysis 
will be presented in a forthcoming publication 14 . 

When the network changes its behavior from homogeneous re- 
trieval to spatial localized state, the critical storage capacity of the 
network drops dramatically, because effectively only the fraction of the 
network in the bump can be excited and the storage capacity drops 
proportionally to the size of the bump, FigEJ As can be seen in this 
figure, in the phase transition point the capacity drops approximately 
twofold. 

The local overlap of the bump remains strong but, in contrast to 
the linear threshold network, for binary neurons is always less than 
one. However, if the threshold is determined as a function of the 
distribution of the internal neuronal fields, as in p^, the impact on the 
storage capacity of the binary network near the transition points is 
minor, as it can be seen by simulations. 
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These effects are interesting and deserves more attention in the 
future. 

5 Conclusion 

In this paper we have studied the conditions for the appearance of 
spatial dependent activity in a binary neural network model. 

The analysis was done analytically, which was possible due to the 
finite number of relevant order parameters, and is compared with sim- 
ulations. The analytical approach gives a closed form for the equations 
describing the different order parameters and permits their analysis. 

It was shown that the presence of the term H a is sufficient for 
the existence of the SAS. Nor asymmetry of the connection, neither 
sparsity of the code are necessary to achieve spatial asymmetric states 
(bumps). In our opinion, this is the main result of the paper. 

The numerical solution of the analytical results, as well as the 
simulations show that when H a = no SAS can be observed. This is a 
good, although not conclusive argument, that the asymmetry between 
the retrieval and the learning states might be a necessary and not only 
sufficient condition for the observation of spatial asymmetry states. 

Detailed analysis of the problem will be presented in a forthcoming 
publication. 
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